Using Students as Experiment Subjects – An Analysis on Graduate and Freshmen Student Data

The question whether students can be used as subjects in software engineering experiments is debated. In order to investigate the feasibility of using students as subjects, a study is conducted in the context of the Personal Software Process (PSP) in which the performance of freshmen students and graduate students are compared and also related to another study in an industrial setting. The hypothesis is that graduate students perform similarly to industry personnel, while freshmen student’s performance differ. A quantitative analysis compares the freshmen’ and graduate students. The improvement trends are also compared to industry data, although limited data access does not allow a full comparison. It can be concluded that very much the same improvement trends can be identified for the three groups. However, the dispersion is larger in the freshmen group. The absolute levels of the measured characteristics are significantly different between the student groups primarily with respect to time, i.e. graduate students do the tasks in shorter time. The data does not give a sufficient answer to the hypothesis, but is a basis for further studies on the issue.